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" The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

• If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the pailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1)0 Responsive to communlcation(s) filed on . 

2a)D This action is FINAL. 2b)S This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex pa/te Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) H Claim(s) 1-93 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) S Claim(s) 1-93 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10) K The drawing(s) filed on 09 September 1999 is/are: a)S accepted or bO objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) 0 The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) ^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (0- 

aMA\\ b)n Some*c)n None of: 

1 Certified copies of the priority documents have been received. 

2.n Certified copies of the priority documents have been received in Application No. . 



3.n Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for % list of the certified copies not received. 

14) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 
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2) lEl Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) M Information Disclosure Statement(s) (PTO-1449) Paper No(s) 6 . 6) □ Other: 
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DETAILED ACTION 
Claim Objections 

1 . Claims 4-9 objected to because of the following informalities: 

In Line 1 of Claim 4, "any one of claim 3" should be corrected to "claim 3". 
In Line 1 of Claim 5, "any one of claim 4" should be corrected to "claim 4". 
In Line 1 of Claim 6, "any one of claim 4" should be corrected to "claim 4". 
In Line 1 of Claim 7, "any one of claim 6" should be corrected to "claim 6". 
Claims 8 and 9 depend on Claims 6 and 7, respectively. 
Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

2. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

3. Claims 1-10, 18-27, 35-44, 52-61, 66-75 and 80-89 are rejected under 35 
U.S.C. 102(b) as being anticipated by Modestino et al (IEEE Paper ISBN: 0162-8828; A 
Markov random Field Model-Based Approach to Image Interpretation). 

Regarding Claim 1, Modestino et al disclose a method of classifying a digital 
image, said method comprising the steps of: 

providing a region adjacency graph representing the digital image (Figure 1; 
Page 607, Column 2, Section MB, first Paragraph); and 

analyzing said region adjacency graph for predetermined patterns and for each 
identified pattern selecting a classification of said digital image (Figures 6(a)-6(c); Page 
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413, Column 1. The digital image is classified based on the identified features as 
depicted in Figure 6(b).). 

Regarding Claim 2, Modestino et al further disclose the method according to 
Claim 1 , wherein said region adjacency graph is classified as one of a plurality of 
stereotypes (Table III (b) and III (c). The region adjacency graph is classified based on 
higher-level expression or stereotypes (e.g., rural road scenes) for the segmentation 
and classification purpose. 

Regarding Claim 3, Modestino et al further disclose the method according to 
Claim 2, wherein a stereotype is assigned to a region adjacency graph on the basis of a 
size of one or more regions of said digital image (Tables l-IV, Areas of the RAG nodes). 

Regarding Claim 4, Modestino et al further disclose the method according to 
Claim 3, wherein a stereotype is assigned to a region adjacency graph on the basis of 
an adjacency of said regions (Figures 1 and 6(b). Region adjacency graphs are 
inherently generated based on the spatial dependence of objects in the relative 
proximity of each other). 

Regarding Claim 5, Modestino et al further disclose the method according to 
Claim 4, wherein a stereotype is assigned to a region adjacency graph on the basis of 
its semantic label content (Page 613, Table lll(b). Stereotypes are assigned to region 
adjacency graph based on the semantic label content, i.e., "road, field" or "sky, field" as 
depicted in table lll(b).). 

Regarding Claim 6, Modestino et al further disclose the method according to 
Claim 4, wherein a stereotype is assigned to a region adjacency graph on the basis of a 
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mean color of one or more regions (Page 613, table lll(a), Average Gray Level of the 
regions). 

Regarding Claim 7, Modestino et al further disclose the method according to 
Claim 6. wherein said plurality of stereotypes are stored in an association lookup table 
(Page 607, Column 2, Section MB, second Paragraph. Set of all interpretation labels L 
is the database or lookup table for stereotypes or node labels.). 

Regarding Claim 8, Modestino et al further disclose the method according to 
Claim 6, wherein said stereotypes are represented in a hierarchal arrangement (Figure 
1. Sky, Road and Car are represented in a hierarchical arrangement. Same concept is 
applicable to the higher-level expressions or node labels in Tables lll(b) and IV(b).). 

Regarding Claim 9, Modestino et al further disclose the method according to 
Claim 7, wherein each of said stereotypes has a hierarchical path arrangement (Figure 
1, Sky, Road and Car has a hierarchical path arrangement. Same concept is 
applicable to the higher-level expressions or node labels in Tables lll(b) and IV(b).). 

Regarding Claim 10, Modestino et al disclose the method according to claim 1, 
wherein said region adjacency graph is provided by analyzing contextual data 
associated with one or more regions of said digital image (Figure 1 , Page 607, Section 
MB, first Paragraph). 

With regards to Claims 18, 35, 52, 66 and 80. arguments analogous to those 
presented for Claim 1 are applicable to Claims 18, 35, 52, 66 and 80. Concerning 
Claims 52, 66 and 80, Modestino et al further disclose providing a set of labeled regions 
(Abstract Lines 5-10; Page 607, Column 2, Section IIB, second paragraph, set of labels 
L = {Li, L2, .... Lm}). Markov Random Field (MRF) model-based approach segments the 



Application/Control Number: 09/395.993 Page 5 

Art Unit: 2623 

image into a collection of disjoint regions that form the nodes of an adjacency graph. 
Once the adjacency graph has been determined, image classification will be achieved 
through assigning object labels to the segmented regions using domain knowledge, 
extracted feature measurements, and spatial relationship between the various regions 
(Abstract, Lines 5-10). 

With regards to Claims 19, 36, 53, 67 and 81 , arguments analogous to those 
presented for Claim 2 are applicable to Claims 19, 36, 53, 67 and 81. 

With regards to Claims 20, 37, 56, 70 and 84, arguments analogous to those 
presented for Claim 3 are applicable to Claims 20, 37, 56, 70 and 84. 

With regards to Claims 21, 38, 55, 69 and 83, arguments analogous to those 
presented for Claim 4 are applicable to Claims 21, 38, 55, 69 and 83. 

With regards to Claims 22, 39, 54, 68 and 82, arguments analogous to those 
presented for Claim 5 are applicable to Claims 22, 39, 54, 68 and 82. 

With regards to Claims 23, 40, 57, 58, 71 , 72, 85 and 86, arguments analogous 
to those presented for Claim 6 are applicable to Claims 23, 40, 57, 58, 71, 72, 85 and 
86. 

With regards to Claims 24, 41, 59, 73 and 87, arguments analogous to those 
presented for Claim 7 are applicable to Claims 24, 41 , 59, 73 and 87. 

With regards to Claims 25, 42, 60, 74 and 88, arguments analogous to those 
presented for Claim 8 are applicable to Claims 25, 42, 60, 74 and 88. 

With regards to Claims 26, 43, 61, 75 and 89, arguments analogous to those 
presented for Claim 9 are applicable to Claims 26, 43, 61, 75 and 89. 
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With regards to Claims 27 and 44, arguments analogous to those presented for 
Claim 10 are applicable to Claims 27 and 44. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 1 1 , 17, 28, 34, 45, 51 , 62-65, 76-79 and 90-93 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Modestino et al in view of Li et al (U.S. 
5,930,783). 

Regarding Claim 11, Modestino et al do not explicitly disclose the method 
according to Claim 10, wherein said contextual data comprises information generated 
by one or more separate sources of said information. 

Li et al disclose a semantic and cognition based image retrieval methodology 
comprising analyzing contextual data generated by one or more separate sources of 
information (Figure IB, Semantic-based Query, Cognition-based Query; Column 12, 
Lines 30-40). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to analyze contextual data generated by one or more separate sources of 
information because it will expand versatility of the image segmentation and 
classification. It will utilize the advantages of image retrieval based on both image 
semantics and visual examples of the image (Li et al, Column 3, Lines 27-35). 
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Regarding Claim 17, Modestino et al do not explicitly disclose the method 
according to Claim 1, wherein said digital image is stored in a database of digital 
images and wherein said classification can be used to retrieve said digital image from 
said database. 

Li et al disclose a semantic and cognition based image retrieval methodology 
wherein the digital image is stored in a database of digital images and wherein the 
classification can be used to retrieve said digital image from the database (Column 4, 
Lines 32-50). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to use classification to retrieve the digital images stored in a database because it 
will utilize a plurality of information sources for image retrieval including semantic- 
based, cognition-based and scene-based queries thereby eliminating weakness of the 
individual approaches (Li et al; Column 3, Lines 27-35). 

With regards to Claims 28 and 45, arguments analogous to those presented for 
Claim 1 1 are applicable to Claims 28 and 45. 

With regards to Claims 34 and 51, arguments analogous to those presented for 
Claim 17 are applicable to Claims 34 and 51. 

With regards to Claims 63, 77 and 91, arguments analogous to those presented 
for Claim 17 are applicable to Claims 63, 77 and 91 . Li et al retrieve the digital image 
by using a keyword representing a stereotype (Column 12, Lines 30-50). 

With regards to Claims 64, 78 and 92, arguments analogous to those presented 
for Claim 17 are applicable to Claims 64, 78 and 92. Neither Modestino nor Li et al 
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explicitly retrieve the digital image by using an icon to represent a stereotype. Utilizing 
an icon to represent a keyword is extremely well known in the art (Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination to Utilize an icon 
for representing a keyword because icons are a significant factor in the user- 
friendliness of graphical user interface that serve as visual mnemonics to allow the user 
to control certain computer actions without having to remember commands or type them 
at the keyboard. 

With regards to Claims 65, 79 and 93, arguments analogous to those presented 
for Claim 17 are applicable to Claims 65, 79 and 93. Li et al retrieve the digital image 
by using a keyword representing a generalization of a stereotype (Column 12, Lines 30- 
50). 

With regards to Claims 62, 76 and 90, arguments analogous to those 
represented for Claim 64 concerning utilizing an icon is applicable to claims 62, 76 and 
90. 

6. Claims 12-16. 29-33 and 46-50 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Modestino et al further in view of Li et al (U.S. 5,930,783) and Jain et 
al (US 6,360,234). 

Regarding Claim 12, neither Modestino et al nor Li et al disclose the method 
according to Claim 11, wherein a corresponding portion of said contextual data is 
obtained from a temporal region of interest for each source of said information. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image wherein a corresponding portion of contextual data is 
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obtained from a temporal region of interest for each source of information (Figures 6-9; 
Column 6, Lines 30-67). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination according to the 
teachings of Jain et al to obtain a corresponding portion of the contextual data from a 
temporal region of interest for each source of information because it will accurately 
classify and intelligently extract information, termed metadata, about the contents of 
video stream in real time (Jain et al, Column 1, Lines 46-49). 

Regarding claim 13, Jain et al further disclose the method according to Claim 12, 
further comprising the step of providing metadata associated with said digital image, 
wherein said metadata includes said stereotypes of said digital image (Figures 16 and 
17; Column 13, Lines 52-67. Column 14, Lines 1-31). 

Regarding Claim 14, Jain et al further disclose the method according to Claim 13, 
wherein said metadata includes a hierarchical path associated with said respective 
stereotype of each digital image (Figures 9, 15-17; Table 1; Column 8, Lines 22-61). 

Regarding Claim 15, Jain et al further disclose the method according to Claim 14, 
wherein said hierarchical path is stored with a respective stereotype as a metadata 
object which is associated with a respective image object (Figure 6; Column 6, Lines 
29-38). 

Regarding Claim 16, Jain et al further disclose the method according to Claim 14, 
wherein said hierarchical path is stored as a referenced lookup table (Figure 7). 

With regards to Claims 29 and 46, arguments analogous to those presented for 
Claim 12 are applicable to Claims 29 and 46. 
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With regards to Claims 30 and 47, arguments analogous to those presented for 
Claim 13 are applicable to Claims 30 and 47. 

With regards to Claims 31 and 48, arguments analogous to those presented for 
Claim 14 are applicable to Claims 31 and 48. 

With regards to Claims 32 and 49, arguments analogous to those presented for 
Claim 15 are applicable to Claims 32 and 49. 

With regards to Claims 33 and 50, arguments analogous to those presented for 
Claim 16 are applicable to Claims 33 and 50. 

Other prior art cited 
7. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

U.S. Patent 5,579,444 to Dalziel et al is cited for an adaptive vision-based 
controller. 

U.S. Patent 6,144.375 to Jain et al is cited for a multi-perspective viewer for 
content-based interactivity. 

IEEE Transaction on Pattern Analysis and Machine Intelligence, ISBN: 0-8186- 
291 0-X to Kim et al is cited for efficient image understanding based on the Markov 
Random Field Model and error backpropagation network. 

IEEE Transaction on Pattern Analysis and Machine Intelligence, ISBN: 0162- 
8828 to Kim et al is cited for an integration scheme for image segmentation and labeling 
based on Markov Random Field Model. 
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8. Any inquiry concerning this comnnunication or earlier communications from the 
examiner should be directed to Mehrdad Dastouri whose telephone number is (703) 
305-2438. The examiner can normally be reached on Monday to Friday from 8:00 a.m. 
to 4:30 p.m. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Amelia Au can be reached on (703) 308-6604. 

The fax phone numbers for the organization where this application or proceeding 
is assigned are (703) 872-9314 for regular and for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
4700. 
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